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This presentation provides an overview of three software tools that were developed by the NASA Glenn 
Research Center to support the assessment of system health: the Propulsion Diagnostic Method 
Evaluation Strategy (ProDIMES), the Systematic Sensor Selection Strategy (S4), and the Extended 
Testability Analysis (ETA) tool. Originally developed to support specific NASA projects in aeronautics and 
space, these software tools are currently available to U.S. citizens through the NASA Glenn Software 
Catalog. 

The ProDiMES software tool was developed to support a uniform comparison of propulsion gas path 
diagnostic methods. Methods published in the open literature are typically applied to dissimilar 
platforms with different levels of complexity. They often address different diagnostic problems and use 
inconsistent metrics for evaluating performance. As a result, it is difficult to perform a one-to-one 
comparison of the various diagnostic methods. ProDiMES solves this problem by serving as a theme 
problem to aid in propulsion gas path diagnostic technology development and evaluation. The overall 
goal is to provide a tool that will serve as an industry standard, and will truly facilitate the development 
and evaluation of significant Engine Health Management (EHM) capabilities. ProDiMES has been 
developed under a collaborative project of The Technical Cooperation Program (TTCP) based on 
feedback provided by individuals within the aircraft engine health management community. 

The S4 software tool provides a framework that supports the optimal selection of sensors for health 
management assessments. S4 is structured to accommodate user-defined applications, diagnostic 
systems, search techniques, and system requirements/constraints. One or more sensor suites that 
maximize this performance while meeting other user-defined system requirements that are presumed 
to exist. S4 provides a systematic approach for evaluating combinations of sensors to determine the set 
or sets of sensors that optimally meet the performance goals and the constraints. It identifies optimal 
sensor suite solutions by utilizing a merit (i.e., cost) function with one of several available optimization 
approaches. As part of its analysis, S4 can expose fault conditions that are difficult to diagnose due to an 
incomplete diagnostic philosophy and/or a lack of sensors. S4 was originally developed and applied to 
liquid rocket engines. It was subsequently used to study the optimized selection of sensors for a 
simulation-based aircraft engine diagnostic system. 


The ETA Tool is a software-based analysis tool that augments the testability analysis and reporting 
capabilities of a commercial-off-the-shelf (COTS) package. An initial diagnostic assessment is performed 
by the COTS software using a user-developed, qualitative, directed-graph model of the system being 
analyzed. The ETA Tool accesses system design information captured within the model and the 
associated testability analysis output to create a series of six reports for various system engineering 
needs. These reports are highlighted in the presentation. The ETA Tool was developed by NASA to 
support the verification of fault management requirements early in the Launch Vehicle process. Due to 
their early development during the design process, the TEAMS-based diagnostic model and the ETA Tool 
were able to positively influence the system design by highlighting gaps in failure detection, fault 
isolation, and failure recovery. 
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NASA Glenn Controls & Dynamics Branch 




Active Clearance Control 

• Clearance Modeling 

• Mechanical/Smart Materials 


System Health Management for 
Propulsion, Power, C Ground Systems 


Maintainability C Reliability 

• Autonomous Mobile Robotic 




Actuation Requirements 

Active Flow Control 

• High Bandwidth Actuation 

• Stall Control 

• Smart Vbnes 

• Turbine Rim Cooling Control 

Active Combustion Control 

• Emission Mnirrization 

• Control of Thermo-aooustic Instability 

Advanced Control Logic 

• Intelligent Adaptive Control 

• Life Extending Control 

• Resilient Propulsion Control 


• Sensor Selection & Validation 

• Fault Dagnosis & Qassifi cation 

• Prognostics 

• Post lest Diagnostic Systems 

• Communication Requirements 

• Real-Time Implementation Issues 


Current NASA Programs 

Aeronautics Research Mission 

• Fundamental Aeronautics 

• Aviation Safety 

Exploration Systems Mission 

• Space Launch System 

• Ground Systems 
Development and Operations 


Inspection & Repair 



Advanced Propulsion Concepts 

• Pulse Detonation Engine 

• Fuel Cell Powered Arcraft 

• VVbve Rotors 

• High-speed Systems 
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Software Tools for System Health Assessment 




ProDIMES: Propulsion Diagnostic 

Method Evaluation Strategy 

Supports fair, quantitative benchmark comparisons 
of aero engine gas path diagnostic methods. 


Iterative Down-Select Process 


Candidate Sensor Suites 


Candidate Selection 
Complete 


S4: Systematic Sensor Selection Strategy 

Software framework for performing optimal selection of 
sensors required to support system health assessment 



System 

Diagnostic 

-1 

Model 

i 


ETA Tool 
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Obtain O/S System Environmental 
Variables and Command Line Inputs 
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I Access TEAMS Testability Oulput 
Dependency Matrix (D-Matrix) 
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| Access Diagnostic Model 

■ Hierarchical Structure File 
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I Access Diagnostic Model 

| Individual Files 
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PerformDetectabilityAnalysis | ) i Gen er ate Rep ort 
Perform Fault Isolation Analysis |— Generate Report 
Perform Test Utilization Analysis |-fc Generate Report 


Perform Component Fault Isolation 
Analysis 


> Generate Report 


t Mapping Analysis | Generate Report 


^Perform Sensor Sensitivity Analysis Generate Report 


^ Application Specific Non-Application Specific 

ETA Tool: Extended Testability 
Analysis Tool 

Testability analyses that support qualitative 
verification of system health requirements 
early in the design process. 
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ProDIMES: Propulsion Diagnostic 

Method Evaluation Strategy 
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Pro pulsion Diagnostic Method Evaluation Strategy (ProDiMES) 

A Public Approach for Benchmarking Gas Path Diagnostic Methods 



Engine Health Management (EHM) related R&D 
activities have increased significantly since the 
late 1990’s. However, due to the use of different 
terminologies, applications, proprietary data, and 
metrics there is no basis of comparison 

Public benchmarking problems can facilitate the 
development and comparison of candidate 
health management methods against a common 
problem 

ProDiMES provides a simulated aircraft engine 
gas path diagnostic benchmarking problem 

- Developed as part of a collaborative project 
under The Technical Cooperation Program 
(TTCP) 

- Available through the NASA Glenn Software 
Catalog 
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Designer 




ProDiMES Architecture 
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Pro pulsion Diagnostic Method Evaluation Strategy (ProDiMES)' 

ProDiMES Diagnostic Benchmarking Process 

ProDiMES enables independent development and evaluation 

and a blind test case comparison 

ProDiMES Public Benchmarking Process 
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la. Engine fleet simulator: Enables user to specify the 
type and number of gas path fault cases. 

lb. Blind test cases: User has no a priori 
knowledge of fault existence or fault type 
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Blind Test Case 
Side-by-Side 
Comparison 


Blind Test Case Data 
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2. Solution providers apply 
their individual diagnostic 
solutions 



Sensed parameter 
histories 
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3. Evaluation Metrics: Defined 
and applied to provide a 
uniform assessment of 
performance 
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Pro pulsion Diagnostic Method Evaluation Strategy (ProDiMES) 

Commercial Modular Aero-Propulsion System Simulation 







Engine model representative of a large commercial turbofan 
NASA-developed, Matlab/Simulink-based, generic, non-linear, 
component-level model with closed-loop control 
Uses 13 health parameters to model engine performance 
degradation due to engine wear and faults. 

Generates 11 sensed outputs 

- 3 aircraft parameters (Pamb, P2, T2) 

- 8 engine measurements (Nf, Nc, P24, Ps30, T24, T30, T48, 
Wf36) 

ProDiMES EFS uses core elements of C-MAPSS: 

- Steady-state solver balances engine to specified operating point 

- No closed-loop control logic or transient operating capability 

- Includes logic to ensure that operating limits are not violated 

- Captures coupled fault effects (e.g., a corrected rotor speed 
sensor fault will result in mis-scheduled variable geometry). 
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Pro pulsion Diagnostic Method Evaluation Strategy (ProDiMES) 

ProDiMES Blind Test Case Comparison Results 



• A 2012 workshop was held for ProDiMES users to share diagnostic results 
and lessons learned. 

- ProDiMES was found to provide a suitably challenging problem. 

- Users welcomed the opportunity to assess and compare diagnostic methods 
against a standard benchmark problem. 


• The results of four diagnostic methods applied to the blind test case data 
set were assessed and compared (a portion of the metric results are shown 
below) 

- Follow-on collaborative assessments conducted by ProDiMES participants have 
shown improved diagnostic performance obtained by pairing the best performing 
detection and classification approaches. 


Abrupt Faults 



NASA Glenn Research Center, Controls & Dynamics Branch, Kevin J. Melcher 


www.nasa.gov 


National Aeronautics and Space Administration 

NASA Glenn Controls & Dynamics Branch 

Software Tools for System Health Assessment 




S4: Systematic Sensor Selection Strategy 


Iterative Down-Select Process 
Candidate Sensor Suites 
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| Application Specific 
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Motivation for Optimizing Sensor Selection 



• Traditional aerospace approaches to sensor selection are: 

- Generally qualitative approaches based on engineering judgment 
and/or heuristics 

- Oriented toward operations, controls and/or performance 

- May not provide coverage required to manage system health 

• Systematic Sensor Selection Strategy (S4) 

- Used in simulation-based studies to optimally identify sensors 
needed to manage the health of liquid rocket engines (MC-1, RS- 
83, RS-84, X-34) and aircraft turbine engines. 

- Model-based approach identifies a set of sensors that optimally 
meets application-specific design objectives 

- Single-valued user-defined metric (i.e., cost) function employed to 
quantitatively assess capability of various sensor combinations to 
meet multiple design objectives 

- Flexible two-stage optimization approach with broad applicability 
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S4 Framework 



Iterative Down-Select Process 
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Generalized S4 Implementation Process 



Step 1 

Knowledge 

Acquisition 


Step 2 

Define 
Sensor 
Suite Merit 
Algorithm 



Step 4 

Choose 

Down- 

Select 

Algorithm 



Step 6 

Develop 

Software 

Modules 




Step 7 

\ 

Step 8 

Create & 

\ 

Create & 

Conduct 


Perform 

Iterative 


Final 

Down- 


Selection 

Select 


Process 

Process 






/ 


- Eight-step process defined as guideline for implementing elements 
of the S4 framework as part of a user-defined application 

- Significant flexibility - early steps can be reordered at user’s 
discretion 

- Some iteration likely during implementation of early steps 

- Detailed description of implementation process contained in 
NASA/C R— 20 12-2 15242 S4 User Guide (see references) 
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S4 Use Case Implementation (1/3) 



Step 1: Knowledge Acquisition 

- Goal: Collect data required to implement user’s S4 application 

- S4 C-MAPSS Use Case 

• Optimization based on seven (7) control sensors and five (5) candidate sensors 

• Data for four (4) fault cases: fan, high pressure compressor, high pressure turbine, 


and low pressure turbine. 

Each fault modeled as simultaneous 
adjustments to associated C-MAPSS 
efficiency and flow capacity health 
parameters. 

Model reference & test case data 

- Used to assess sensor suite 
failure detection and fault isolation. 

- Data conditioning improves 
algorithm stability 


Low Pressure Compressor 


High Pressure Compressor 


High Pressure Turbine 

Low Pressure Turbine 



Low Speed Shaft 
High Speed Shaft 


® Control Sensors Bvpass 
O Candidate Sensors 


Combustor 
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S4 Use Case Implementation (2/3) 



Step 2: Define Sensor Suite Merit Algorithm 


- Goal: Define algorithm that will be used to quantify optimality 

- S4 C-MAPSS Use Case 


For each sensor suite k 



= P k D k 


where 


1 


Wl 


N d ~N k 


+ 1 


• 2 % 


7=1 


• M k Merit Value - Value of the Merit Algorithm for sensor suite k. 

• P k Penalty Term - Reduces merit value (M k ) as number of sensors in 

suite k deviates from the preferred number of sensors, N d = 9. 

• D k Diagnostic Performance Score - Provides quantitative assessment 

of the diagnostic performance of sensor suite k. 

• Cj Criticality Factor - Weighting term for fault case j based on fault 

criticality and/or probability of occurrence. For this use case = 1 . 

• Z jk Fault Diagnostic Performance Metric - Qualitative assessment of 

the diagnostic performance of sensor suite k for fault case j. 
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S4 Use Case Implementation (3/3) 



Steps 7 & 8: Create/Perform Iterative Down-Select & Final 
Selection Processes 

- Goal: Develop and execute software code for processes that perform the 

optimization. 

- S4 C-MAPSS Use Case 

• Iterative Down-Select Process 

- Implemented with Genetic Algorithm 

- After 5 generations, highest performing 
sensor suites are identified 

• Final Selection Process 

- Selected 3 highest performing sensor 
suites for more rigorous evaluation 

— Confirms results of Iterative Down-Select Process 



Sensor Suite Merit 
Value 

Add’l 

Candidate 

Sensors 

Iterative 

Down-Select 

Process 

Final 

Selection 

Process 

T50, P50 

23.0418 

22.4238 

P25, T50 

22.0360 

20.8636 

P25, P50 

22.0360 

20.8636 
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ETA Tool 


Command Line Input 




ETA Tool: Extended Testability 
Analysis Tool 
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Model-based Testability Analysis 



Model-based Testability Analysis 

- Qualitative diagnostic approach 
based on a fault propagation model 
of the system 

- Generates Dependency Matrix that 
relates failure modes to failure 
detection tests 


Motivation 

- Multiple system-level goals for 
Safety, Reliability, Maintainability and 
Availability lead to various diagnostic- 
related requirements levied against 
the system or subsystem. 

Benefits 


Approach 

- TEAMS Designer (commercial-off- 
the-shelf software) used to model 
fault propagation via directed graph 
theory. 


DIRECTED GRAPH 


Faults propagate in 
forward direction and 
are detected by “tests’ 
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- Model-based verification of diagnostic 
requirements early in the design 
process enables redesign when cost 
impacts are lower. 
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ETA Tool Overview 



Qualitative Functional Fault Model 
with Testability Analysis 
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□ 


Gov’t Dev. Effort 


Commercialization Effort 




Stand-alone software tool developed Government/Industry partnership 
to support Ares Program integrated ETA tool modeling 

Provides additional analyses and conventions, analyses, and reports 
detailed reporting capabilities ' n t° existing COTS software 

package to support future NASA 

Failure Detectability j programs. 
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Stand-alone software available from the NASA Glenn Software Catalog at: 

https://sr.qrc.nasa.qov 
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ETA Tool Use Case 

Fault Isolation Report Detail 


• Failure modes are grouped by their “detection signature” - the set of tests that detect the 
propagated effects 

• Fault isolation information can be used to verify Caution & Warning or Launch Commit 
Criteria (LCC) strategies when detection must identify a specific set of failure modes. 


Failure Mode Group #22 


Detection Signature: 3 Tests 


Sensor 

Identifier 

Schematic 

Identifier 

Sensor Description 

Test 

SEN0026 

D4 

Yaw LVDT Displacement Transducer 1 

No Position Change 

SEN0027 

D5 

Yaw LVDT Displacement Transducer 2 

No Position Change 

SEN0028 

D 6 

Yaw LVDT Displacement Transducer 3 

No Position Change 


Failure Mode Group contains 2 Failure Mode(s) 


System 

Component 

Failure Mode 

FMEA Identifier 

Criticality 

Vector Control System 

Power Valve 

Stuck In NULL Position 

MS-SS-ACT-02-002 

1 

Vector Control System 

Yaw Actuator 

Actuator Locked In Place 

MS-SS-ACT-03-001 

1 
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ETA Tool Use Case 

Fault Isolation Report Summary 


Ambiguity Analysis 


Number of Failure Mode Croups 61 

Number of Failure Modes 1 1 8 

Number of Isolated Groups 50 

Maximum Failure Mode Group Size 1 5 

Calculated Ambiguity Score 55 


Ambiguity Distribution {Failure Mode Level) 

Percentage of groups with 1 members 81 .97% (50 of 61 groups) 

Percentage of groups with 2 members 6.56% ( 4 of 61 groups) 

Percentage of groups with 3 members 3.28% ( 2 of 61 groups) 

Percentage of groups with 6 members 3.28% ( 2 of 61 groups) 

Percentage of groups with 10 members 1 .64% ( 1 of 61 groups) 

Percentage of groups with 15 members 3.28% ( 2 of 61 groups) 


Failure Mode Ambiguity Distribution 


Number of Failure 
Mode Groups 
(6 i Total) 


Group Size i 2 3 4 5 6 7 3 9 10 11 12 13 14 15 
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Concluding Remarks 


• NASA Glenn developed software tools available to 
U.S. Citizens through the NASA Glenn Software 
Catalog/Repository (http://sr.grc.nasa.gov). 

• Software Catalog includes software tools developed 
by the Controls and Dynamics Branch to support the 
assessment of System Health. 

- ProDIMES provides approach for fair, quantitative benchmark 
comparisons of aero engine gas path diagnostic methods. 

- S4 provides software framework for optimally selecting sensors 
required to support the assessment of system health. 

- ETA Tool augments commercial-off-the-shelf software to 
provide testability analyses that support qualitative verification 
of system health requirements early in the design process. 

• For more information contact kjmelcher@nasa.gov 
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